Add support for multiple S3 targets per backup job - #492

Why

A backup job can only be uploaded to a single S3 endpoint. Mirroring a backup to a second provider currently means either running a second job that dumps the same database again, or a hand written post script that reimplements the upload, the checksum handling and the retention.

What

S3_TARGETS takes a comma separated list of target names and uploads the very same dump and checksum file to each of them, so all endpoints hold identical content.

DEFAULT_S3_TARGETS: hetzner,impossible
DEFAULT_S3_PATH: db                                    ## shared by both targets

S3_HETZNER_HOST: fsn1.your-objectstorage.com
S3_HETZNER_BUCKET: backups
S3_HETZNER_KEY_ID: xxxxxxxx
S3_HETZNER_KEY_SECRET: xxxxxxxx

S3_IMPOSSIBLE_HOST: eu-central-2.storage.impossibleapi.net
S3_IMPOSSIBLE_BUCKET: backups-offsite
S3_IMPOSSIBLE_KEY_ID: yyyyyyyy
S3_IMPOSSIBLE_KEY_SECRET: yyyyyyyy

Every S3 option exists per target as S3_<TARGET>_<OPTION> and per job as DB01_S3_<TARGET>_<OPTION>. The value used for a target is the first one that is set of:

DB01_S3_<TARGET>_<OPTION> → S3_<TARGET>_<OPTION> → DB01_S3_<OPTION> → S3_<OPTION> → DEFAULT_S3_<OPTION>

so shared settings are written once and only what differs is repeated per target. _FILE secrets work for the per target variables as well.

Behaviour

• Backwards compatible. With S3_TARGETS unset the target list holds a single empty entry and the existing S3_ variables are used, exactly as before.
• Every target is attempted, including after an earlier one failed, so a broken endpoint no longer keeps the healthy ones from receiving their backup. A failure names the target in the log and is aggregated into move_exit_code, which already drives the failure notification and reaches post scripts as $11.
• The S3 retention pass loops over the same targets and now builds its own aws-cli environment per target, instead of relying on the credentials, endpoint URL and TLS flags left behind by the preceding upload.
• An unreachable endpoint is retried by aws-cli before it gives up, which can stall a job for minutes. This is documented, with S3_<TARGET>_EXTRA_OPTS: --cli-connect-timeout 10 --cli-read-timeout 30 as the remedy.

Testing

Verified with the released container image against two MinIO endpoints using different bucket names:

• identical sha256 for the dump on both targets, checksum file on both
• one endpoint stopped: logged as Failed moving backup to S3 Bucket - target 'minio2' - exit code 1 while the healthy target still received the backup
• retention deleting on both targets, leaving both with the same set of files
• an unchanged single target configuration still working

Also included: README documentation, a examples/multiple-s3-targets/compose.yml, and a CHANGELOG entry.

Note

This branch is independent of my two other PRs and can be merged in any order.